Specification-Based Testing of Intrusion Detection Systems

An Intrusion Detection System (IDS) protects computer networks against attacks and intrusions, in combination with firewalls and anti-virus systems. An IDS is therefore a crucial element of a network security posture. One class of IDS is called signature-based network IDSs as they monitor network traffic, looking for evidence of malicious behavior as specified in attack descriptions (referred to as signatures). Many studies have reported that IDSs can generate thousands of alarms a day thus overwhelming administrators with false alarms. It is therefore important to precisely understand under which conditions IDSs accurately identify attacks and under which conditions they fail to do so or raise false alarms. Four approaches are generally discussed to evaluate signature-based IDSs: (1) Using vulnerability exploitation programs, i.e., programs often available on the Internet, that actually attack systems; (2) Using an IDS stimulator that relies on the specification of the IDS (i.e., its signatures) to create (sequence of) packet(s) to attack systems; (3) Using Honeypots (vulnerable systems) that capture live attacks from real attackers on the Internet; (4) Using a live network (e.g., in an organization) to study the IDS behavior on normal traffic. In this paper, we investigate the IDS stimulator strategy and precisely define a number of test adequacy criteria based on IDS specifications (signatures). We experiment with one criterion on one widely used and maintained IDS and show that our test approach is effective at revealing numerous defects.